Fibre Channel is a switched communications protocol that allows concurrent communication among servers, workstations, storage devices, peripherals, and other computing devices. Fibre Channel can be considered a channel-network hybrid, containing enough network features to provide the needed connectivity, distance, and protocol multiplexing, and enough channel features to retain simplicity, repeatable performance, and reliable delivery. Fibre Channel is capable of full-duplex transmission of frames at rates extending from 1 Gbps (gigabits per second) to 10 Gbps. It is also able to transport commands and data according to existing protocols such as Internet protocol (IP), Small Computer System Interface (SCSI), High Performance Parallel Interface (HIPPI) and Intelligent Peripheral Interface (IPI) over both optical fiber and copper cable.
FIG. 1 illustrates a variable-length Fibre Channel frame 10. The frame 10 has a 4-byte start-of-frame (SOF) indicator 12, which is a particular binary sequence indicative of the beginning of the frame 10. The SOF indicator 12 is followed by a 24-byte header 14, which specifies, among other things, the frame source address and the frame destination address. A variable-length data field 16 follows the header 14, which can range from 0 to 2112 bytes in length. The data field 16 is followed by a 4-byte cyclical redundancy check (CRC) code 18 for error detection, and by a 4 byte end-of-frame (EOF) indicator 20. Since the data payload 16 of a Fibre Channel frame can vary between 0 and 2112 bytes, the total length of a Fibre Channel frame 10 can vary from 36 to 2148 bytes.
The frame header 14 is twenty-four bytes long, containing six four-byte words as shown in FIG. 2. The frame header 14 contains a destination address or identifier (D_ID) 60 and a source address or identifier (S_ID) 70. These address fields 60, 70 both contain a twenty-four bit address that uniquely identifies a device on a Fibre Channel fabric, and are described in more detail below. In addition to these address fields 60, 70, the frame header 14 contains a routing control field (R_CTL) 40, a class specific control field (CS_CTL) 42, frame type (TYPE) 44, frame control field (F_CTL) 46, sequence identifier (SEQ_ID) 48, data field control (DF_CTL) 50, sequence count (SEQ_CNT) 52, originator exchange identifier (OX_ID) 54, responder exchange identifier (RX_ID) 56, and parameter value (PARAM) 58. The R_CTL field 40 contains routing bits and categorizes the frame. The CS_CTL 42 contains class specific control data or priority data. The TYPE field 44 identifies the protocol of the Fibre Channel frame 10. The F_CTL field 46 contains control information relating to the frame content, such as first sequence, last sequence, or end sequence bits. The SEQ_ID field 48 identifies frames within a sequence to assure frame order, and to correlate link control frames with related data frames. The DF_CTL field 50 specifies the presence of optional headers at the beginning of the data payload 16. The SEQ_CNT field 52 indicates the sequential order of the data frame 10 within a single sequence or multiple consecutive sequences for the same exchange. The OX_ID field 54 identifies the data communication exchange for the originating computer. The RX_ID field 56 identifies the data communication exchange for the originally receiving computer. The parameter/data field 58 can identify link control frames and carry related data. Alternatively, for data frames, the parameter/data field 64 can contain a data offset value. The third bit of the F_CTL field 46 indicates whether the parameter/data field 64 is being used to indicate a relative offset or not.
As explained above, the destination identifier (D_ID) 60 identifies the desired destination for the frame, while the source identifier (S_ID) 70 identifies the source for the frame. These identifiers 60, 70 are considered port identifiers or Fibre Channel addresses 80, as shown in FIG. 3. Port identifiers 80 are uniquely assigned to every node in a Fibre Channel fabric. Under the standard Fibre Channel switch fabric addressing scheme, each port identifier 80 is considered to contain three eight-bit words: a domain address or domain ID 82, an area address or area ID 84, and a port address or port ID 86. Each switch in a Fibre Channel fabric is generally assigned a unique domain address 82. When switch ports are directly connected to only a single device, the area ID 84 will representing a logical or physical grouping of ports, with the particular port being identified in by port ID 86. More frequently, however, multiple devices are allowed to connect to a single physical port on a switch using an arbitrated loop. In these circumstances, the area ID 84 represents a particular port on the switch, and the port ID 86 identifies the particular device on the loop. The addressing scheme allows 256 port IDs 86 and 256 area IDs 84, but only 239 domain IDs 82. This smaller number of domain IDs results from the fact that some domain/switch addresses are reserved.
Fibre Channel switches use the destination identifier 60 found in header 14 to route frames 10 from a source port to a destination port. The Fibre Channel addressing scheme in FIG. 3 allows certain routing decisions to be made by examining only the eight bits in the domain ID 82. Typically, this is accomplished using a lookup table at each input port. The destination identifier 60 is used as an index to the table, and the table returns the appropriate output port in the switch. This output port will either be directly connected to the node identified by the destination identifier 60, or to another switch along the path to the identified destination. Routing tables are shared between multiple switches in a fabric over an interswitch link so that the switches can learn about the nodes and switches that make up the fabric.
Switches uses port identifiers 80 to establish zoning for a fabric, which allows devices to be logically segregated based on function, business departments, or operating system conflicts. FIG. 4 shows an example of zoning, where servers A and B exist on the same fabric 90 as disk devices X and Y, and tape storage Z. Disk devices X and Y are assigned to zones 1 and 2, respectively, while tape storage Z is assigned to zone 3. Server A is part of zones 1 and 3, and therefore is allowed by the fabric 90 to communicate with disk device X and tape storage Z. Server B is part of zones 2 and 3, meaning that this server is allowed to communicate only with disk device Y and tape storage Z. In this manner, zoning prevents server A from seeing or accessing disk device Y, and server B from seeing or accessing disk device X. Both servers are allowed access to tape storage Z. Zoning can be done on a port basis, thereby allowing a particular physical port on a switch to communicate with a different physical port elsewhere on the fabric. Unfortunately, this type of zoning is meaningless if the devices are ever physically moved from one port on a switch to another.
Alternatively, zoning can occur at the World Wide Name level. A World Wide Name (or WWN) is a 64-bit name that is uniquely assigned to every Fibre Channel device and port created. By using the WWN to define and implement the zones, a particular computer or storage device will be recognized by its WWN and properly zoned regardless of the physical port on which the device is found.
The Word Wide Name of a device is made known to the fabric during device login, during which a port identifier 80 is also automatically assigned to the device. The Domain ID 82 of the assigned port identifier 80 will be the same as the Domain ID 82 of the switch to which the device is connected. This port identifier 80 is associated with the WWN in a name server operating on the fabric 90. Devices log into this name server during a fabric and port login procedure, which is required before the device can access (or be accessed by) the fabric. The name server records the WWN of the device, and the device's associated port identifier 80. The name server functionality is implemented in each switch on a fabric 90.
Fibre Channel is typically implemented in a fabric made up of one or more Fibre Channel switches. The purpose of the fabric is to interconnect the various devices or node ports (N_ports). This is accomplished by attaching the N_ports to fabric ports (F_ports) associated with a switch in the fabric. Multiple switches communicate with each to form a multi-switch fabric. Switch to switch communication links takes place over expansion ports (E_ports), and are generally referred to as inter-switch links.
The interswitch links formed over the E_ports are used to coordinate knowledge about the physical layout of the fabric, which is used when routing Fibre Channel frames 10 over a fabric 90. Switches use a link state database and a routing protocol known as Fabric Shortest Path First (FSPF) to make routing decisions over a fabric 90. To communicate this information, Fibre Channel switches are able to communicate a variety of message types, including extended link service messages (FC-ELS), generic service messages (FC-GS), and switch internal link service messages (SW-ILS).
Some of the more important extended link services (ELS) are N_Port Device Discovery: FLOGI—Fabric Login, PLOGI—N_Port Login, and RSCN—Registered State Change Notification. FLOGI enables a device or N_Port to obtain its port identifier 80 and to create a communication channel with a switch by setting or exchanging various operational parameters. PLOGI enables a device to create a communication channel with the remote device over the fabric by setting and exchanging operational parameters. An RSCN is issued by Fabric Controller in a switch to notify all the registered devices when an event occurs, such as a failed connection with a device or a new device.
The generic services include services performed by the name server in the switch. For instance, an RFF_ID is a communication sent by a device to a switch after the device completes FLOGI. This allows the device to be registered with the name server database. The switch then uses an RSCN ELS to notify other registered devices of this new device.
The switch internal link services (SW-ILS or ILS) are used to configure the Switch Fabric, and to build and maintain required routing tables. Some of the most important ILS commands include ELP (Exchange Link Parameters), HLO (an FSPF Hello), LSU (an FSPF Link State Update), LSA (an FSPF Link State Acknowledgement), and SW_RSCN (an Inter-Switch RSCN). The ELP helps to establish a communication channel between adjacent E_Port by setting or exchanging operational parameters. All of these ILS commands are issued between adjacent E_Ports and use Fabric Controller Fibre Channel address 80 (0xFFFFFD) as the S_ID and D_ID for both request and reply sequences. The HLO command establishes a communication channel between two switches between their E_ports. LSU and LSA commands contain Link State Records (LSR) and Link Descriptors (LD), which are used to establish and maintain a synchronized topology database for the fabric and a related routing table. Finally, a SW_RSCN is used to distribute ELS RSCNs between switches.
In order for Fibre Channel routing to function properly, it is vital that every device on a fabric 90 have a unique port identifier 80 and that every switch in the fabric 90 have a different domain ID 82 assigned to it. This effectively limits the size of a network to at most 237 switches. Although this is a large number of switches for most circumstances, this limit requires the physical separation of one fabric from another. It would not be possible to create a world-wide storage area network similar to the Internet given a limitation of 237 switches.
This reality has recently led several vendors to explore the possibility of sharing data between separate fabrics. Unfortunately, the requirement for unique port identifiers 80 has made it impossible to transmit frames between fabrics without considerable concern over the port identifiers involved in the communication. Since each fabric separately assigns port identifiers 80, it is likely that there will be some overlap in assigned domain IDs 82 and port identifiers 80 between the fabrics.
One approach to this problem is set forth in U.S. Published Patent Application No. 2005/0025075. In this application, the separate fabrics are described as Virtual SANs. While virtual SANs are storage area networks that are virtually defined as separate even though they exist on the same physical switch, the technique of this application could be used to communicate across separate physical fabrics. The problem of conflicting domain IDs is avoided by implementing techniques to ensure this doesn't happen. For instance, the application suggests administratively partitioning the domain number space across the VSANs to be connected so that each fabric can assign only a unique subset of domain names. Alternatively, the application suggests reserving a subset of domain IDs to be used only for switches involved in inter-VSAN communications. In this way, both fabrics could be allowed to use most of the available domain IDs 82 (such as the first 200) in a standard manner without concern over duplicate usage. The remaining 37 domain IDs would be assigned from a central location, thereby ensuring only one switch in the combined fabric would be assigned a particular domain ID 82. All communication between fabrics would then be limited to those devices whose domain IDs 82 are centrally managed.
A second approach is outlined in U.S. Published Patent Application No. 2003/0012204. In this application, multiple fabrics are linked together using gateways so as to allow the combined storage area networks to exceed the Fibre Channel limit of 237 switches in a single fabric. This application overcomes the problem of conflicting Domain IDs by assigning each switch a “global address” as well as the local address typically assigned in Fibre Channel fabrics. The global address is created by combining multiple physical switches into a virtual switch, and then assigning a single global domain ID to the virtual switch group. When communications are made using this global address, the entire set of switches is considered a single virtual switch. The ports on the various switches are then addressed directly as ports on the virtual switch. Internal communications within the SAN use only local addressing, and are unaffected by the presence of the gateways and the related global addresses. When a communication is directed to a device on a separate remote fabric, the communication will use the global address for that device, using the virtual switch ID associated with that device. The routing for this communication passes through the local or ingress gateway. The ingress gateway translates the source address in the communication from the local address created by the local device to the pre-assigned virtual address (with the virtual address typically combining multiple physical switches into a single virtual switch). The ingress gateway then transmits this communication on to a remote or egress gateway physically connected to the remote fabric. The egress gateway looks to see if the D_ID is a virtual address (to one of its virtual switches). If so, the egress gateway translates the D_ID to a local address. Virtual addresses for a device only make sense outside of the physical SAN on which the device is located. Although each fabric is still limited to 237 switch addresses, the combining of switches on remote fabric into virtual switches means that many more than 237 switches can be accessed on the combined inter-fabric.
Unfortunately, neither of the above approaches provides a flexible enough technique so as to allow inter-fabric data communication between Fibre Channel fabrics.